The advent of global communication networks such as the Internet has facilitated widespread dissemination of information through corporate enterprise(s) in the form of disparate databases typically utilized by separate corporate entities or branch-offices at various geographic locations. Merging such heterogeneous data sources into a homogeneous database heretofore poses a significant system-intensive process that results in duplication of many system processes.
Disparate databases can be converged by means of replication—replication is a process that copies and distributes data and database objects from one database to another, and then synchronizes information between the databases for consistency.
Merge replication is a difficult replication type. Merge replication capability enables disconnected and mobile users to run applications offline, and then periodically reconnect to synchronize with a main database. It makes possible autonomous changes to replicated data on a source (also called a publisher) and on a destination (also called a subscriber) while connected or disconnected, and then merging the updates between sites when the sites are connected. With merge replication, a server captures incremental data changes in the source database and in the destination database, and reconciles conflicts according to preconfigured rules or via employing a custom resolver to resolve the conflicts.
Merge replication is typically used in support of autonomous changes of replicated data at the source and/or destination. The data is synchronized between servers either at a scheduled time or on demand. Updates are made independently (e.g., no commit protocol) at more than one server; so same data can be updated by the source or by more than one destination. Consequently, conflicts can occur when data modifications are merged. Merge replication includes default and custom choices for conflict resolution that can be defined when a merge source is configured. When a conflict occurs, a merge agent invokes the conflict resolver to determine which data will be accepted and propagated to other destination sites. Options available with merge replication include filtering source data horizontally and vertically, including employment of join filters and dynamic filters, using alternate synchronization partners, optimizing synchronization to improve merge performance, validating replicated data to ensure synchronization, and utilizing attachable subscription databases.
Merge replication supports horizontal, dynamic, and join filtering, all of which enable an administrator to create partitions (or tables) of data to be replicated. Filtering of replicated data allows for at least the following: a minimization of the amount of data sent over a network; reduction in the amount of storage space required at a destination replica; customization of data sources and applications based upon individual destination replica requirements; and avoidance or mitigation of conflicts because the respective data partitions can be sent to different destination replicas. Even though merge replication allows multiple replicas to update the same data, filtering of the data such that replicas receive disjoint sets, ensures that no two replicas to a single destination will be updating the same data values.
Traditionally, merge replication has supported techniques to keep a partition of a source replica consistent with a destination replica. However, existing algorithms introduce significant performance degradation of a system by requiring concurrent synchronization sessions between the source and destination replicas. In conventional servers, merge replication has had to support very sophisticated partitioning techniques to allow destination replicas to receive only a subset of the data of the source replica. These techniques require a high number of CPU cycles at the source replica, and thus create a bottleneck for expanding a network by increasing the number of simultaneous synchronizations required for maintaining destination database replication. Intensive hardware and software processor functions are associated with “partition computation” algorithms that prepare a list of changes needed to keep the partition of the destination replica consistent with that of the source replica.
Under a conventional partition computation approach, when a client requests database synchronization with a server, the partition computation function of the server includes computing the partition membership of rows that have not been synchronized in order to determine differences between the two databases. Since the client database is typically a subset of an overall database residing on the server, the computation function further includes reviewing substantially all current changes at the source replica that may potentially need to be propagated to the destination replica, and then determine which of those changes are relevant to the destination replica. For instance, rows that qualify according to the destination replica filter criteria belong to the destination replica and must be updated or inserted at the destination replica. Rows that have undergone updating and no longer satisfy the filer criteria, no longer belong to the destination replica and hence, must be deleted. This implies that an update at the source replica could be propagated as a delete operation to the destination replica. Further, rows that have undergone updating and now belong to the destination replica may need to be expanded, if there are join filters, to obtain the related rows from the joined tables.
Some negative aspects from a performance and scaling point of view of the “per synchronization session partition computation” approach are as follows. Conventional systems are burdened with redundant processing, since for every change, irrespective of whether there is partition update or not, a conventional synchronization session needs to evaluate whether the changed row belongs to the destination replica or not. If a row changes before each synchronization session, every subsequent synchronization session evaluates again, whether that row belongs to the destination replica, by establishing new partition membership metadata for that row. The knowledge about a row's previous partition membership is not remembered between synchronization sessions. This leads to redundant processing for each synchronization session. To illustrate the process intensive aspects of the conventional system by example, consider a network with one thousand destination replicas. If a row is determined by the source to have undergone a change and requires publication to the destinations, the one thousand destination replicas synchronize to be more up-to-date. If the row changes again, another one thousand synchronization sessions follow, and this process is repeated until the row has changed one thousand times. In the end, all of the synchronization sessions will have executed a combined total of one million partition computation functions for the same row.
Conventional systems exhibit poor concurrency behavior; since for every row change, each destination replica is required to perform its own evaluation. In the presence of multiple simultaneous synchronizations between the source and destination systems, there will be multiple instances of the partition computation executing simultaneously, causing high CPU consumption in the source server.
The partition computation algorithm that prepares the list of changes needed to keep the partition of the destination replica consistent with that of the source replica tends to be the most CPU and query processor intensive functions that are executed in any given synchronization session. If the filtering used to define the set of replicated data were not optimal, the performance of concurrent partition computation functions would further degrade the capabilities of the source server.
Faster hardware and more efficient replication configuration can address this issue to a certain degree. But even in a perfectly designed and tuned application, every Publisher server has an upper limit with respect to the number of simultaneous Subscriber merge processes it can handle effectively.
One solution to this problem is to coordinate the timing of the source merge processes so that the processes are staggered. If all destinations currently connect at the beginning or end of their work day to merge changes, merge processes can be required to be performed at off-peak times during the day. If there is no control over when destinations perform merge processes, an upper limit can be defined to limit the number of destination that are allowed to merge at the same time.
Another way to reduce the demands on a source server is to spread processing load across more sources. A common technique is to use several servers in a republishing hierarchy. For example, if a single server currently acts as a source for all sales representatives in the country, two more source servers could be added to distribute the load associated therewith, wherein a Central Publisher publishes data to East and West Subscribers, an East Subscriber republishes data to the sales representatives on the East coast, and a West Subscriber republishes data to the sales representatives on the West coast.
However, none of the conventional solutions provide a cost effective and efficient architecture for addressing the problem associated with a large number of synchronizations. Conventional systems propagate irrelevant metadata over the network. Since there is no persisted membership metadata even after synchronization sessions have computed partitions, all changes at the source replica will be considered relevant for propagation to the destination replica. This approach contributes to the scaling problem, since the list of changes to be considered grows if a large number of destination replicas propagate changes to the source replica. Moreover, if the source replica records any additional metadata during one synchronization session with a destination replica, there might be a need to propagate this metadata to other destination replicas. Hence, the scaling characteristics of the current approach are less than desirable. There is a need in the art to minimize the amount of metadata that is considered and propagated over the network during the synchronization session. Based upon deficiencies in the prior art to support many concurrent synchronizations sessions, there is a need to eliminate expensive partition computation functions during the execution of every synchronization session.